Sky replacement

ABSTRACT

The present disclosure provides systems and methods for image editing. Embodiments of the present disclosure provide an image editing system for perform image object replacement or image region replacement (e.g., an image editing system for replacing an object or region of an image with an object or region from another image). For example, the image editing system may replace a sky portion of an image with a more desirable sky portion from a different replacement image. The original image and the replacement image (e.g., the image including a desirable object or region) include layers of masks. A sky from the replacement image may replace the sky of the image to produce an aesthetically pleasing composite image.

BACKGROUND

The following relates generally to image editing, and more specificallyto replacing regions or objects of an image with portions of anotherimage.

Image editing is a process used to alter properties of an image, forexample, to increase the quality of an image or video. In some cases, animage is altered to have a desired appearance or to improve thevisibility or clarity of the image. Replacing portions of an image andinserting portions of one image into another are common image editingtasks.

For example, in some cases users may wish to replace the sky in oneimage with the sky from another image. Sky replacement and other regionreplacement tasks can be performed by image editing softwareapplications. However, transferring portions of one image into anotherimage can be difficult and time-consuming. For example, using currenttechniques, the photographer first manually identifies a sky region anda foreground region with labels or brushes. Once identified, the sky andforeground regions can be segmented from each other.

In some existing cases, the identification and segmentation of thedifferent regions may include manually assigning individual pixels a skyor non-sky label. This manual segmenting process is done for both theimage having the desired foreground region and the image having thedesired sky region. Segmentation problems may arise due to a number offactors including large variations in appearance, and complicatedboundaries with other regions or objects such as trees. That is, smallportions of a sky may be located between leaves of a tree, which canmake segmentation challenging or inaccurate.

Thus, users who wish to replace regions of an image are often faced witha tedious and time-consuming task. Furthermore, conventional imageediting applications may create unwanted visual artefacts and unnaturaleffects when inserting content from one image into another. Therefore,there is a need in the art for an improved image editing applicationthat can efficiently combine images in a natural looking way.

SUMMARY

The present disclosure provides systems and methods for image editing.Embodiments of the present disclosure provide an image editing systemfor perform image object replacement or image region replacement (e.g.,an image editing system for replacing an object or region of an imagewith an object or region from another image). For example, the imageediting system may replace a sky portion of an image with a moredesirable sky portion from a different sky replacement image (e.g., asky preset image).

A method, apparatus, non-transitory computer readable medium, and systemfor image editing are described. Embodiments of the method, apparatus,non-transitory computer readable medium, and system are configured togenerate a plurality of region masks for a first image using a maskgeneration network, wherein the region masks correspond to a samesemantically related region of the first image, generate a defringinglayer by combining a first region mask of the plurality of region maskswith grayscale version of a second image, generate a region-specificlayer by combining a second region mask of the plurality of region maskswith the second image, and generate a composite image by combining thefirst image, the defringing layer, and the region-specific layer.

A method, apparatus, non-transitory computer readable medium, and systemfor image editing are described. Embodiments of the method, apparatus,non-transitory computer readable medium, and system are configured togenerate a first region mask, a second region mask, and a third regionmask corresponding to a same semantically related region of a firstimage, generate a defringing layer by combining the first region maskwith grayscale version of the second image, generate a region-specificlayer by combining the second region mask and the third region mask toproduce a combined region mask, and combining the combined region maskwith the second image, and generate a composite image by combining thefirst image, the defringing layer, and the region-specific layer.

An apparatus, system, and method for image editing are described.Embodiments of the apparatus, system, and method include a maskgeneration network configured to generate a plurality of region masksfor a first image using a mask generation network, wherein the regionmasks correspond to a same semantically related region of the firstimage, a defringing component configured to generate a defringing layerby combining a first region mask of the plurality of region masks withgrayscale version of a second image, a region-specific layer componentconfigured to generate a region-specific layer by combining a secondregion mask of the plurality of region masks with the second image, anda layer composition component configured to generate a composite imageby combining the first image, the defringing layer, and theregion-specific layer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of an image editing system according to aspectsof the present disclosure.

FIG. 2 shows an example of a process for image editing according toaspects of the present disclosure.

FIG. 3 shows an example of an image editing apparatus according toaspects of the present disclosure.

FIG. 4 shows an example of a preset component according to aspects ofthe present disclosure.

FIG. 5 shows an example of a horizon adjustment process according toaspects of the present disclosure.

FIG. 6 shows an example of a region layer flowchart according to aspectsof the present disclosure.

FIG. 7 shows an example of a region layer diagram according to aspectsof the present disclosure.

FIG. 8 shows an example of a lighting layer flowchart according toaspects of the present disclosure.

FIG. 9 shows an example of a lighting layer diagram according to aspectsof the present disclosure.

FIGS. 10 through 11 show examples of a process for generating acomposite image according to aspects of the present disclosure.

FIG. 12 shows an example of a process for color harmonization accordingto aspects of the present disclosure.

FIGS. 13 through 14 show examples of a process for image editingaccording to aspects of the present disclosure.

FIG. 15 shows an example of a preset image diagram according to aspectsof the present disclosure.

FIG. 16 shows an example of a preset image representation structureaccording to aspects of the present disclosure.

DETAILED DESCRIPTION

The present disclosure provides systems and methods for image editing.Embodiments of the present disclosure provide an image editing systemfor performing image object replacement or image region replacement(e.g., an image editing system for replacing an object or region of animage with an object or region from another image). In some embodiments,one or more region masks are generated based on an original image. Theregion masks are combined with portions of a source image in multiplelayers, and the layers are combined to produce a composite image.

In an example scenario, a photographer may perform a photoshoot on anovercast or rainy day. However, the overcast sky may not be desirablefor an aesthetically pleasing image. Therefore, the photographer mayprefer another sky from another picture taken during a sunny day.However, the process of replacing a sky portion of the image involvessegmenting the foreground of the image from the sky, replacing the skywith a sky from another image, and performing color matching to make thecomposite image natural looking. This process can be difficult andtime-consuming.

Using current techniques, a photographer first manually identifies thesky region and foreground region with labels or brushes in a designapplication. Once identified, the sky and foreground regions can besegmented from each other. In some cases, the identification andsegmentation of the different regions may include manually assigningindividual pixels a sky or non-sky label. These and other segmentationproblems arise due to a number of factors including large variations inimage appearance and complicated boundaries with other regions orobjects such as trees, mountains, water, and the like. Such techniquescan be tedious and time-consuming, and in some cases result in unwantedvisual artefacts and unnatural effects in generated composite images.

Embodiments of the present disclosure include an improved imagereplacement process that automatically segments the foreground andbackground of an image (i.e., the original image) using one or moreimage segmentation masks. In some examples, two region masks aregenerated that have varying levels of sharpness at the boundary (e.g.,the boundary between the foreground and the background. These masks arecombined to form a composite mask that achieves the appropriate level ofboundary sharpness. Adjustment options, such as a brush tool, a fadeedge, and a shift edge may be used to combine the masks.

In some embodiments, a defringing layer is generated by combining aregion mask of the original image with a grayscale version of thereplacement image. The defringing layer is combined with the originalimage and a region specific layer (e.g., a layer that reveals areplacement background from another image) to produce a composite image.Accordingly, the produced composite image may include the foreground ofthe original image and the background of the replacement image, but thelighting of the composite image near the region boundary is morerealistic.

Embodiments of the present disclosure provide a fully automatic,non-destructive image replacement method using the original image and areplacement image as input. A layer structure enables thenon-destructive effect. That is, the replacement effect is provided byadding layers on top of an original image rather than editing the imageitself. Embodiments of the present disclosure use an advanced pipelinefor mask editing in a preview mode.

Additionally, real-time color harmonization based on the visible skyregion of the replacement image may be used to produce more naturalcolorization. In some examples, horizon-aware sky alignment andplacement with advanced padding may also be used. For example, thehorizons of the original image and the replacement image may beautomatically detected and aligned. Additionally, some embodiments ofthe present disclosure provide fast preset loading and saving.

Embodiments of the present disclosure may be used in the context of atool for replacing a sky region of one image with a sky region ofanother image. A sky replacement example is provided with reference toFIGS. 1 through 5 . Details regarding generating a region replacementlayer (e.g., a sky layer) are provided with reference to FIGS. 3 and 4 .Examples of a defringing layer are provided with reference to FIGS. 8through 10 . Details regarding color harmonization layer are providedwith reference to FIGS. 11 and 12 . Details regarding preset loading areprovided with reference to FIGS. 13 through 16 .

Sky Replacement

FIG. 1 shows an example of an image editing system according to aspectsof the present disclosure. The example shown includes user 100, device105, cloud 110, image editing apparatus 115, and database 120. Imageediting apparatus 115 is an example of, or includes aspects of, thecorresponding element described with reference to FIG. 3 .

The image editing system uses compositing methods based on combiningimages and segmentation masks to replace a sky portion of an image whilesuppressing artefacts such as a halo/fringing effect, to automaticallycreate high-quality sky replacement results.

Some compositing techniques such as alpha matting result in unwantedvisual artefacts. Even if the segmentation is perfect, differencesbetween the images being composited (e.g., color/brightness differencesbetween the foreground from one image and the sky from another) may leadto objectionable fringing artefacts. This can be mitigated by blurringout the segmentation, but such blurring may lead to halo artefacts.

Therefore, in the example of FIG. 1 , the user 100 interacts with animage editing apparatus 115 via a device 105. In some examples, theimage editing apparatus 115 is located on the cloud 110. However, insome examples the image editing apparatus 115 is located on the device105 itself. The user 100 provide an original image such as a picture orphotograph. In the example shown, the original image includes a buildingin rainy weather. The user 100 then finds an image with a more appealingbackground (i.e., from among images stored on the device 105 or withinthe database 120).

The image editing apparatus 115 segments the original image to identifya foreground and a background. Then, the image editing apparatus 115creates a composite image using the foreground of the original image anda sky region from the selected replacement image. Multiple segmentationmasks may be combined with the sky replacement image to create a skyreplacement layer, and one of these segmentation masks (or a separatemask) may also be combined with a greyscale version of the replacementsky to create a defringing layer (e.g., which in some case may bereferred to as a lighting layer) that reduces the halo effect. In someembodiments, the image editing apparatus 115 automatically harmonize thecolors of the foreground and the new background (i.e., the replacementsky).

Embodiments of the present disclosure utilize compositing methods thatgenerate high-quality composites without fringing and halo artefacts. Insome examples, the methods described herein generate multiplemasks—e.g., a hard mask and a soft mask—and use carefully selectedblending modes to combine the masks. This method may be performedautomatically and works well across many sky replacement examples. Insome cases, the input to the system includes a single image and a presetreference image is applied for the replacement sky. In otherembodiments, a user 100 selects two images and a composite is made fromthe two images. In some cases, the system automatically selects portionsof the images for composition (i.e., it can identify and replace a skyregion automatically). According to embodiments of the presentdisclosure, a layer structure is used that enables non-destructiveeffects.

The device 105 may be a personal computer, laptop computer, mainframecomputer, palmtop computer, personal assistant, mobile device, or anyother suitable processing apparatus. The device 105 may include imageediting software. The image editing software can include a variety ofediting tools including the sky replacement system described in FIG. 1 .

In some examples, a device 105 may also include an optical instrument(e.g., an image sensor, camera, etc.) for recording or capturing images,which may be stored locally, transmitted to another location, etc. Forexample, an image sensor may capture visual information using one ormore photosensitive elements that may be tuned for sensitivity to avisible spectrum of electromagnetic radiation. The resolution of suchvisual information may be measured in pixels, where each pixel mayrelate an independent piece of captured information. In some cases, eachpixel may thus correspond to one component of, for example, atwo-dimensional (2D) Fourier transform of an image. Computation methodsmay use pixel information to reconstruct images captured by the device.

A cloud 110 is a computer network configured to provide on-demandavailability of computer system resources, such as data storage andcomputing power. In some examples, the cloud 110 provides resourceswithout active management by the user 100. The term cloud is sometimesused to describe data centers available to many users 100 over theInternet. Some large cloud networks have functions distributed overmultiple locations from central servers. A server is designated an edgeserver if it has a direct or close connection to a user 100. In somecases, a cloud 110 is limited to a single organization. In otherexamples, the cloud is available to many organizations. In one example,a cloud 110 includes a multi-layer communications network comprisingmultiple edge routers and core routers. In another example, a cloud 110is based on a local collection of switches in a single physicallocation.

According to some embodiments, database 120 stores preset informationincluding the region location information and the low-resolution imagedata for each of the set of preset images in a same preset informationfile. In some examples, database 120 stores high-resolution image datafor each of the preset images in separate image files. In some examples,the separate image files includes JPEG or PNG files. According to someembodiments, database 120 may be configured to store a presetinformation file including the region location information and thelow-resolution image data in a preset information file and to storehigh-resolution image data for the preset images in separate imagefiles.

The image editing apparatus 115 includes a computer implemented networkthat generates a composite image. In some embodiments, the image editingapparatus 115 includes a mask generation network, a defringingcomponent, a region-specific layer component, a layer compositioncomponent, a color harmonization component, an image property component,an image editing application, and a preset component. The presetcomponent will be described with reference to FIG. 4 . The image editingapparatus 115 takes an original image and a replacement image to producea composite image.

The image editing apparatus 115 may also include a processor unit, amemory unit, and a user interface. Additionally, image editing apparatus115 can communicate with the database 120 via the cloud 110. Furtherdetail regarding the architecture of the image editing apparatus 115 isprovided with reference to FIGS. 3 and 4 .

In some cases, the image editing apparatus 115 is implemented on aserver. A server provides one or more functions to users 100 linked byway of one or more of the various networks. In some cases, the serverincludes a single microprocessor board, which includes a microprocessorresponsible for controlling all aspects of the server. In some cases, aserver uses microprocessor and protocols to exchange data with otherdevices/users on one or more of the networks via hypertext transferprotocol (HTTP), and simple mail transfer protocol (SMTP), althoughother protocols such as file transfer protocol (FTP), and simple networkmanagement protocol (SNMP) may also be used. In some cases, a server isconfigured to send and receive hypertext markup language (HTML)formatted files (e.g., for displaying web pages). In variousembodiments, a server comprises a general purpose computing device, apersonal computer, a laptop computer, a mainframe computer, asupercomputer, or any other suitable processing apparatus.

A database 120 is an organized collection of data. For example, adatabase 120 stores data in a specified format known as a schema. Adatabase 120 may be structured as a single database, a distributeddatabase, multiple distributed databases, or an emergency backupdatabase. In some cases, a database controller may manage data storageand processing in a database 120. In some cases, a user 100 interactswith database controller. In other cases, database controller mayoperate automatically without user 100 interaction.

In some instances, the present disclosure uses the terms original image,foreground image, and target image interchangeably. For example, anoriginal image may be the target for replacement with the sky from areplacement image (i.e., the source of the sky). In the example of FIG.1 , the original image, sent by the user 100 to the image editingapparatus 115, may include a foreground region (e.g., the building) anda target replacement region (e.g., the overcast sky, which the user 100intends to replace via the image editing apparatus 115).

The terms replacement image, background image, preset image, sourceimage, reference image, sky image, and sky preset may also be usedinterchangeably. In the example of FIG. 1 , the replacement image,generated by the image editing apparatus 115 and displayed or returnedto user 100, may include a background region (e.g., a sunny sky), whichin some cases may also be referred to as a source region, a sky region,etc.

FIG. 2 shows an example of a process for image editing according toaspects of the present disclosure. In some examples, these operationsare performed by a system including a processor executing a set of codesto control functional elements of an apparatus. Additionally oralternatively, certain processes are performed using special-purposehardware. Generally, these operations are performed according to themethods and processes described in accordance with aspects of thepresent disclosure. In some cases, the operations described herein arecomposed of various substeps, or are performed in conjunction with otheroperations.

Embodiments of the present disclosure utilize compositing methods thatgenerate high-quality composites without fringing and halo artefacts. Insome examples, the methods described herein generate a stack of layers—ahard sky mask, a soft sky mask, a lighting mask, and lightingcontent—and use carefully selected blending modes to combine them andsuppress the halo/fringing. This method may be performed automaticallyand works well across many sky replacement examples.

Accordingly, a method for image editing is described. Embodiments of themethod generate a first region mask, a second region mask, and a thirdregion mask corresponding to a same semantically related region of afirst image. Embodiments of the method are further configured togenerate a defringing layer by combining the first region mask withgrayscale version of the second image, generate a region-specific layerby combining the second region mask and the third region mask to producea combined region mask, and combine the combined region mask with thesecond image. Embodiments of the method are further configured togenerate a composite image by combining the first image, the defringinglayer, and the region-specific layer.

At operation 200, the user provides an original image to the imageediting apparatus. In some cases, the operations of this step refer to,or may be performed by, a user as described with reference to FIG. 1 .In some cases, the input to the system includes a single image and apreset reference image is applied. In other embodiments, a user selectstwo images, and a composite is made from the two images. In some cases,the system automatically selects portions of the images for composition(i.e., it can identify and replace a sky region automatically). Theoriginal image and the replacement image can be in any of multipleformats (e.g., PNG, JPEG, RAW, etc.).

Furthermore, according to embodiments of the present disclosure, a layerstructure is used that provides the non-destructive effect. Someconventional image editing applications only generate a replaced picture(e.g., a new picture with the sky replaced). However, according toembodiments of the present disclosure, an image may be replaced with acomposite picture including different layers (i.e., pixel and adjustmentlayers) with layer masks, which allows users to further adjust theindividual component to their preferences (and to undo any unwantedchanges).

At operation 205, the system displays preset previews. Thumbnail data ofthe presets are loaded for preview. The full resolution data of a presetmay not be loaded into the memory until it is selected by the user forsky replacement. The user may preview the presets before selecting. Insome cases, the operations of this step refer to, or may be performedby, an image editing apparatus as described with reference to FIGS. 1and 3 .

At operation 210, the user selects a preset image. When a user selects apreset, the regions in the preset image may be detected to find thecoordinates for placing a new image into the composition. In some cases,the operations of this step refer to, or may be performed by, a user asdescribed with reference to FIG. 1 .

At operation 215, the system loads a high resolution version of the skyimage. In some examples, the high resolution image of a preset image maynot be loaded into the database until it is selected by the user forimage replacement. In such cases, when a new image preset is selectedfor image replacement, the high resolution data of the previouslyselected image preset may be released from the database. In some cases,the operations of this step refer to, or may be performed by, an imageediting apparatus as described with reference to FIGS. 1 and 3 .

At operation 220, the system generates sky replacement layers. In somecases, the operations of this step refer to, or may be performed by, animage editing apparatus as described with reference to FIGS. 1 and 3 .The layer structure may include a layer for the original image, a skyreplacement layer (i.e., based on a combination of a foreground maskfrom the original image and a sky region from the replacement image), adefringing layer (based on a mask from the original image and agrayscale of the replacement sky), and a color harmonization layer. Thelayer structure is described further with reference to FIG. 5 .

At operation 225, the system generates a composite image by combiningthe layers.

Additionally, an apparatus for performing the method is described. Theapparatus includes a processor, memory in electronic communication withthe processor, and instructions stored in the memory. The instructionsare operable to cause the processor to generate a first region mask, asecond region mask, and a third region mask corresponding to a samesemantically related region of a first image, generate a defringinglayer by combining the first region mask with grayscale version of thesecond image, generate a region-specific layer by combining the secondregion mask and the third region mask to produce a combined region mask,and combining the combined region mask with the second image, andgenerate a composite image by combining the first image, the defringinglayer, and the region-specific layer.

A non-transitory computer readable medium storing code for image editingis described. In some examples, the code comprises instructionsexecutable by a processor to: generate a first region mask, a secondregion mask, and a third region mask corresponding to a samesemantically related region of a first image, generate a defringinglayer by combining the first region mask with grayscale version of thesecond image, generate a region-specific layer by combining the secondregion mask and the third region mask to produce a combined region mask,and combining the combined region mask with the second image, andgenerate a composite image by combining the first image, the defringinglayer, and the region-specific layer.

A system for image editing is described. Embodiments of the system areconfigured for generating a first region mask, a second region mask, anda third region mask corresponding to a same semantically related regionof a first image, generating a defringing layer by combining the firstregion mask with grayscale version of the second image, generating aregion-specific layer by combining the second region mask and the thirdregion mask to produce a combined region mask, and combining thecombined region mask with the second image, and generating a compositeimage by combining the first image, the defringing layer, and theregion-specific layer.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include applying a mask brushto adjust the combined region mask. Some examples further includeapplying a fade edge adjustment, a shift edge adjustment, or both thefade edge adjustment and the shift edge adjustment after applying themask brush.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include applying the maskbrush to further adjust the combined region mask. Some examples furtherinclude automatically reapplying the fade edge adjustment, the shiftedge adjustment, or both the fade edge adjustment and the shift edgeadjustment after reapplying the mask brush.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include generating a colorharmonization layer based on the second region mask, wherein thecomposite image includes the color harmonization layer.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include adjusting a positionof the second image relative to the first image. Some examples furtherinclude automatically regenerating the defringing layer and theregion-specific layer based on the position. Some examples furtherinclude automatically regenerating the composite image based on theregenerated defringing layer and the regenerated region-specific layer.

In some examples, the first region mask has a more gradual mask boundarythan the second region mask, and the second region mask has a moregradual boundary than the third mask. In some examples, the semanticallyrelated region of the first image comprises a first sky region, and thecomposite image comprises a second sky region from the second image.

FIG. 3 shows an example of an image editing apparatus 300 according toaspects of the present disclosure. In one embodiment, image editingapparatus 300 includes user interface 305, processor unit 310, memoryunit 315, mask generation network 320, defringing component 325,region-specific layer component 330, layer composition component 335,color harmonization component 340, image property component 345, imageediting application 350, and preset component 355. Image editingapparatus 300 is an example of, or includes aspects of, thecorresponding element described with reference to FIG. 1 .

An apparatus for image editing is described. Embodiments of theapparatus include a mask generation network 320 configured to generate aplurality of region masks for a first image using a mask generationnetwork, where the region masks correspond to a same semanticallyrelated region of the first image. Embodiments of the apparatus furtherinclude a defringing component 325 configured to generate a defringinglayer by combining a first region mask of the plurality of region maskswith grayscale version of a second image. Embodiments of the apparatusfurther include a region-specific layer component 330 configured togenerate a region-specific layer by combining a second region mask ofthe plurality of region masks with the second image. Embodiments of theapparatus further include a layer composition component 335 configuredto generate a composite image by combining the first image, thedefringing layer, and the region-specific layer.

A user interface 305 may enable a user to interact with a device. Insome embodiments, the user interface 305 may include an audio device,such as an external speaker system, an external display device such as adisplay screen, or an input device (e.g., remote control deviceinterfaced with the user interface 305 directly or through aninput/output (IO) controller module). In some cases, a user interface305 may be a graphical user interface 305 (GUI).

A processor unit 310 is an intelligent hardware device, (e.g., ageneral-purpose processing component, a digital signal processor (DSP),a central processing unit (CPU), a graphics processing unit (GPU), amicrocontroller, an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a programmable logic device, adiscrete gate or transistor logic component, a discrete hardwarecomponent, or any combination thereof). In some cases, the processor isconfigured to operate a memory array using a memory controller. In othercases, a memory controller is integrated into the processor. In somecases, the processor is configured to execute computer-readableinstructions stored in a memory to perform various functions. In someembodiments, a processor includes special purpose components for modemprocessing, baseband processing, digital signal processing, ortransmission processing.

Examples of a memory unit 315 include random access memory (RAM),read-only memory (ROM), or a hard disk. Examples of memory devicesinclude solid state memory and a hard disk drive. In some examples,memory is used to store computer-readable, computer-executable softwareincluding instructions that, when executed, cause a processor to performvarious functions described herein. In some cases, the memory contains,among other things, a basic input/output system (BIOS) which controlsbasic hardware or software operation such as the interaction withperipheral components or devices. In some cases, a memory controlleroperates memory cells. For example, the memory controller can include arow decoder, column decoder, or both. In some cases, memory cells withina memory store information in the form of a logical state.

According to some embodiments, user interface 305 displays a set oflow-resolution previews including the at least one low-resolutionpreview. In some examples, user interface 305 receives feedbackindicating the preset image, where the preset image is selected based onthe feedback. According to some embodiments, user interface 305 may beconfigured to display the at least one low-resolution preview and toreceive feedback for selecting a preset image from among the pluralityof preset images. User interface 305 is an example of, or includesaspects of, the corresponding element described with reference to FIG. 4.

According to some embodiments, mask generation network 320 generates aset of region masks for a first image using a mask generation network320, where the region masks correspond to a same semantically relatedregion of the first image. In some examples, mask generation network 320combines the second region mask with a third region mask of the set ofregion masks to create a combined region mask, where the region-specificlayer is generated using the combined region mask. In some examples, thefirst region mask has a more gradual mask boundary than the secondregion mask. In some examples, the semantically related region of thefirst image includes a first sky region, and the composite imageincludes a second sky region from the second image.

According to some embodiments, mask generation network 320 generates afirst region mask, a second region mask, and a third region maskcorresponding to a same semantically related region of a first image. Insome examples, mask generation network 320 applies a mask brush toadjust the combined region mask. In some examples, mask generationnetwork 320 applies a fade edge adjustment, a shift edge adjustment, orboth the fade edge adjustment and the shift edge adjustment afterapplying the mask brush. In some examples, mask generation network 320applies the mask brush to further adjust the combined region mask. Insome examples, mask generation network 320 automatically reapplies thefade edge adjustment, the shift edge adjustment, or both the fade edgeadjustment and the shift edge adjustment after reapplying the maskbrush. In some examples, the first region mask has a more gradual maskboundary than the second region mask, and the second region mask has amore gradual boundary than the third mask. In some examples, thesemantically related region of the first image includes a first skyregion, and the composite image includes a second sky region from thesecond image.

According to some embodiments, mask generation network 320 may beconfigured to generate a plurality of region masks for a first imageusing a mask generation network 320, wherein the region masks correspondto a same semantically related region of the first image.

In some examples, the mask generation network 320 includes aconvolutional neural network (CNN). A CNN is a class of neural networkthat is commonly used in computer vision or image classificationsystems. In some cases, a CNN may enable processing of digital imageswith minimal pre-processing. A CNN may be characterized by the use ofconvolutional (or cross-correlational) hidden layers. These layers applya convolution operation to the input before signaling the result to thenext layer. Each convolutional node may process data for a limited fieldof input (i.e., the receptive field). During a forward pass of the CNN,filters at each layer may be convolved across the input volume,computing the dot product between the filter and the input. During thetraining process, the filters may be modified so that they activate whenthey detect a particular feature within the input.

In some embodiments, one or more components of the image editingapparatus 300 may include (or implement) one or more aspects of anartificial neural network (ANN). An ANN is a hardware or a softwarecomponent that includes a number of connected nodes (i.e., artificialneurons), which loosely correspond to the neurons in a human brain. Eachconnection, or edge, transmits a signal from one node to another (likethe physical synapses in a brain). When a node receives a signal, itprocesses the signal and then transmits the processed signal to otherconnected nodes. In some cases, the signals between nodes comprise realnumbers, and the output of each node is computed by a function of thesum of its inputs. Each node and edge is associated with one or morenode weights that determine how the signal is processed and transmitted.

During the training process, these weights are adjusted to improve theaccuracy of the result (i.e., by minimizing a loss function whichcorresponds in some way to the difference between the current result andthe target result). The weight of an edge increases or decreases thestrength of the signal transmitted between nodes. In some cases, nodeshave a threshold below which a signal is not transmitted at all. In someexamples, the nodes are aggregated into layers. Different layers performdifferent transformations on their inputs. The initial layer is known asthe input layer and the last layer is known as the output layer. In somecases, signals traverse certain layers multiple times.

In some embodiments, multiple neural networks are used for generatingsegmentation masks. For example, in one embodiment, the mask generationnetwork 320 may include a model for generating a base mask, one forrefining the base mask, and one for detecting difficult regions (e.g.,trees or wires). All of the models may be used to generate the followingmasks: a hard mask (e.g., for the region replacement layer), a soft mask(e.g., for the region replacement layer), and a lighting/defringinglayer.

According to some embodiments, mask generation network 320 generates aforeground region mask for a first image using a mask generation network320. The mask generation network 320 may be configured to generate aforeground region mask for a first image. In some examples, the maskgeneration network 320 includes a convolutional neural network.

According to some embodiments, defringing component 325 generates adefringing layer by combining a first region mask of the set of regionmasks with grayscale version of a second image. In some examples, thedefringing layer is located between the first image and theregion-specific layer. In some examples, defringing component 325adjusts a position of the second image relative to the first image.According to some embodiments, defringing component 325 generates adefringing layer by combining the first region mask with grayscaleversion of the second image. In some examples, defringing component 325adjusts a position of the second image relative to the first image. Insome examples, defringing component 325 automatically regenerates thedefringing layer and the region-specific layer based on the position.

According to some embodiments, defringing component 325 may beconfigured to generate a defringing layer by combining a first regionmask of the plurality of region masks with grayscale version of a secondimage. According to some embodiments, defringing component 325 adjusts aposition of the second image relative to the first image. In someexamples, defringing component 325 generates a defringing layer based ongrayscale data of the second image and the output of the mask generationnetwork 320, where the composite image further includes the defringinglayer. According to some embodiments, defringing component 325 may beconfigured to generate a defringing layer based on an output of the maskgeneration network 320 and grayscale data of the second image.

According to some embodiments, region-specific layer component 330generates a region-specific layer by combining a second region mask ofthe set of region masks with the second image. According to someembodiments, region-specific layer component 330 generates aregion-specific layer by combining the second region mask and the thirdregion mask to produce a combined region mask, and combining thecombined region mask with the second image. According to someembodiments, region-specific layer component 330 may be configured togenerate a region-specific layer by combining a second region mask ofthe plurality of region masks with the second image. In some examples,the region-specific layer component 330 includes a brush tool, a fadeedge slider, and a shift edge slider.

According to some embodiments, region-specific layer component 330generates a background region layer based on the second image and anoutput of the mask generation network 320, where the composite imageincludes the first image, the color harmonization layer, and thebackground region layer. According to some embodiments, region-specificlayer component 330 may be configured to generate a background regionlayer based on an output of the mask generation network 320 and thesecond image.

According to some embodiments, layer composition component 335 generatesa composite image by combining the first image, the defringing layer,and the region-specific layer. In some examples, layer compositioncomponent 335 automatically regenerates the composite image based on theadjusted position. According to some embodiments, layer compositioncomponent 335 generates a composite image by combining the first image,the defringing layer, and the region-specific layer. In some examples,layer composition component 335 automatically regenerates the compositeimage based on the regenerated defringing layer and the regeneratedregion-specific layer. According to some embodiments, layer compositioncomponent 335 may be configured to generate a composite image bycombining the first image, the defringing layer, and the region-specificlayer.

According to some embodiments, color harmonization component 340generates a color harmonization layer based on at least one of the setof region masks, where the composite image includes the colorharmonization layer. According to some embodiments, color harmonizationcomponent 340 generates a color harmonization layer based on the secondregion mask, where the composite image includes the color harmonizationlayer. According to some embodiments, color harmonization component 340may be configured to generate a color harmonization layer based on atleast one of the plurality of region masks.

According to some embodiments, color harmonization component 340generates a color harmonization layer based on the foreground propertydata and the background property data. In some examples, colorharmonization component 340 automatically adjusts the colorharmonization layer based on the adjusted position. In some examples,color harmonization component 340 automatically adjusts the colorharmonization layer based on the adjusted colors. In some examples,color harmonization component 340 computes a set of color harmonizationcurves, where the color harmonization layer is generated based on thecolor harmonization curves. In some examples, the color harmonizationlayer is located between the first image and the background regionlayer. In some examples, the color harmonization layer applies colorsfrom a background portion of the second image to a foreground portion ofthe first image.

According to some embodiments, color harmonization component 340generates a color harmonization layer based on the foreground propertydata and the background property data. In some examples, colorharmonization component 340 automatically adjusts the colorharmonization layer based on the change. In some examples, the colorharmonization layer applies colors from a background portion of thesecond image to a foreground portion of the first image.

According to some embodiments, color harmonization component 340 may beconfigured to generate a color harmonization layer based on theforeground property data and the background property data. In someexamples, the color harmonization component 340 is configured to detecta change in the background property data and automatically adjust thecolor harmonization layer based on the change.

According to some embodiments, image property component 345 computesforeground property data based on the foreground region mask and thefirst image. In some examples, image property component 345 computesbackground property data based on a second image. In some examples, thebackground property data is computed based on an output of the maskgeneration network 320. According to some embodiments, image propertycomponent 345 computes foreground property data based on a foregroundregion mask and a first image. In some examples, image propertycomponent 345 computes background property data based on a second image.In some examples, image property component 345 detects a change in theforeground property data or the background property data. In someexamples, the change includes a position change of the second image withrespect to the first image. In some examples, the change includes achange in color of the second image. In some examples, the changeincludes a change in scale of the second image.

According to some embodiments, image property component 345 may beconfigured to compute foreground property data based on the first imageand background property data based on a second image.

According to some embodiments, image editing application 350 edits thecomposite image using an image editing application 350. According tosome embodiments, image editing application 350 generates a compositeimage based on the first image, the second image, and the colorharmonization layer. In some examples, image editing application 350automatically adjusts the composite image based on the adjusted colorharmonization layer. In some examples, image editing application 350adjusts colors of the second image. In some examples, image editingapplication 350 automatically adjusts the composite image based on theadjusted color harmonization layer. In some examples, the compositeimage replaces a first sky region of the first image with a second skyregion from the second image.

According to some embodiments, image editing application 350 generates acomposite image based on the first image, the second image, and thecolor harmonization layer. According to some embodiments, image editingapplication 350 may be configured to generate a composite image based onthe first image, the second image, and the color harmonization layer.

Preset component 355 is an example of, or includes aspects of, thecorresponding element described with reference to FIG. 4 . According tosome embodiments, preset component 355 receives original image data. Insome examples, preset component 355 retrieves preset information for setof preset images, where the preset information for each of the presetimages includes low-resolution image data and region locationinformation. In some examples, the region location informationcorresponds to a sky region of the preset image. According to someembodiments, preset component 355 loads the preset information from thepreset information file. In some examples, preset component 355 selectsa preset image from among the set of preset images based on the presetinformation. In some examples, preset component 355 loads thehigh-resolution image data for the preset image from one of the separateimage files based on the selection. In some examples, the presetinformation further includes image metadata.

The described systems and methods may be implemented or performed bydevices that include a general-purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof. A general-purpose processor may be amicroprocessor, a conventional processor, controller, microcontroller,or state machine. A processor may also be implemented as a combinationof computing devices (e.g., a combination of a DSP and a microprocessor,multiple microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration). Thus, the functionsdescribed herein may be implemented in hardware or software and may beexecuted by a processor, firmware, or any combination thereof. Ifimplemented in software executed by a processor, the functions may bestored in the form of instructions or code on a computer-readablemedium.

Computer-readable media includes both non-transitory computer storagemedia and communication media including any medium that facilitates thetransfer of code or data. A non-transitory storage medium may be anyavailable medium that can be accessed by a computer. For example,non-transitory computer-readable media can comprise random access memory(RAM), read-only memory (ROM), electrically erasable programmableread-only memory (EEPROM), compact disk (CD) or other optical diskstorage, magnetic disk storage, or any other non-transitory medium forcarrying or storing data or code.

Also, connecting components may be properly termed computer-readablemedia. For example, if code or data is transmitted from a website,server, or other remote source using a coaxial cable, fiber optic cable,twisted pair, digital subscriber line (DSL), or wireless technology suchas infrared, radio, or microwave signals, then the coaxial cable, fiberoptic cable, twisted pair, DSL, or wireless technology are included inthe definition of medium. Combinations of media are also included withinthe scope of computer-readable media.

A system for image editing is described. The system comprises a maskgeneration network configured to generate a plurality of region masksfor a first image using a mask generation network, wherein the regionmasks correspond to a same semantically related region of the firstimage, a defringing component configured to generate a defringing layerby combining a first region mask of the plurality of region masks withgrayscale version of a second image, a region-specific layer componentconfigured to generate a region-specific layer by combining a secondregion mask of the plurality of region masks with the second image, anda layer composition component configured to generate a composite imageby combining the first image, the defringing layer, and theregion-specific layer.

A method of manufacturing an apparatus for image editing is described.The method provides a mask generation network configured to generate aplurality of region masks for a first image using a mask generationnetwork, wherein the region masks correspond to a same semanticallyrelated region of the first image, a defringing component configured togenerate a defringing layer by combining a first region mask of theplurality of region masks with grayscale version of a second image, aregion-specific layer component configured to generate a region-specificlayer by combining a second region mask of the plurality of region maskswith the second image, and a layer composition component configured togenerate a composite image by combining the first image, the defringinglayer, and the region-specific layer.

A method of using an apparatus for image editing is described. Themethod uses a mask generation network configured to generate a pluralityof region masks for a first image using a mask generation network,wherein the region masks correspond to a same semantically relatedregion of the first image, a defringing component configured to generatea defringing layer by combining a first region mask of the plurality ofregion masks with grayscale version of a second image, a region-specificlayer component configured to generate a region-specific layer bycombining a second region mask of the plurality of region masks with thesecond image, and a layer composition component configured to generate acomposite image by combining the first image, the defringing layer, andthe region-specific layer.

In some examples, the mask generation network comprises a CNN. Asdescribed herein, a CNN is a class of neural network that is commonlyused in computer vision or image classification systems. In someexamples, the region-specific layer component comprises a brush tool, afade edge slider, and a shift edge slider. Some examples of theapparatus, system, and method described above further include a colorharmonization component configured to generate a color harmonizationlayer based on at least one of the plurality of region masks.

FIG. 4 shows an example of a preset component 400 according to aspectsof the present disclosure. Preset component 400 is an example of, orincludes aspects of, the corresponding element described with referenceto FIG. 3 . In one embodiment, preset component 400 includessegmentation network 405, thumbnail component 410, preview component415, user interface 420, and color conversion component 425.

According to some embodiments, segmentation network 405 performs asegmentation operation on the high-resolution image data to produce theregion location information. In some examples, the region locationinformation includes bounding box information. According to someembodiments, segmentation network 405 performs a segmentationinformation on each of a set of preset images to produce region locationinformation. In some examples, the region location information includesbounding box information. According to some embodiments, segmentationnetwork 405 may be configured to generate region location informationfor a plurality of preset images. In some examples, the segmentationnetwork 405 includes a CNN.

According to some embodiments, thumbnail component 410 generates atleast one low-resolution preview based on the original image data andthe preset information. According to some embodiments, thumbnailcomponent 410 generates low-resolution image data for each of the presetimages. According to some embodiments, thumbnail component 410 may beconfigured to generate low-resolution image data for the preset images.

According to some embodiments, preview component 415 selects a presetimage from among the set of preset images based on the at least onelow-resolution preview image. In some examples, the low-resolutionpreview includes at least one region of the corresponding preset imagecombined with at least one region of the original image data. Accordingto some embodiments, preview component 415 may be configured to generateat least one low-resolution preview based on the preset information.

User interface 420 is an example of, or includes aspects of, thecorresponding element described with reference to FIG. 3 .

According to some embodiments, color conversion component 425 loadshigh-resolution image data for the selected preset image. In someexamples, color conversion component 425 performs color conversion onthe high-resolution image data based on the original image data.According to some embodiments, color conversion component 425 performscolor conversion on the preset images to produce the high-resolutionimage data. According to some embodiments, color conversion component425 may be configured to perform color conversion on the preset imagesto produce the high-resolution image data.

FIG. 5 shows an example of a horizon adjustment process according toaspects of the present disclosure. In one embodiment, layers panel 500includes sky layer display 505, foreground lighting layer display 510,foreground color layer display 515, and original image layer 520.

Embodiments of the present disclosure provide tools to allow users toadjust a mask both globally (with a fade edge slider) and locally (i.e.,with brush tool) to adjust the masks in different modes with precisecontrol on finer details. In conjunction with the layers, an embodimentprovides editing controls where users may adjust the composite image. Insome examples, an automatic method initializes the control parameters,and the user can fine-tune them. In the example of FIG. 5 , a layerspanel 500 may include sky layer display 505, foreground lighting layerdisplay 510, foreground color layer display 515, and original imagelayer 520, which may display corresponding image editing aspectsdescribed herein.

Region Layer

FIG. 6 shows an example of a region layer flowchart according to aspectsof the present disclosure. In some examples, these operations areperformed by a system including a processor executing a set of codes tocontrol functional elements of an apparatus. Additionally oralternatively, certain processes are performed using special-purposehardware. Generally, these operations are performed according to themethods and processes described in accordance with aspects of thepresent disclosure. In some cases, the operations described herein arecomposed of various substeps, or are performed in conjunction with otheroperations.

At operation 600, the system identifies a first region mask and a secondregion mask for a first image, where the first region mask indicates asemantically related portion of the image, and where the second regionmask indicates the semantically related portion of the image with asofter boundary than the first region mask. In some cases, theoperations of this step refer to, or may be performed by, a maskgeneration network as described with reference to FIG. 3 .

At operation 605, the system identifies a reference region from a secondimage. In some cases, the operations of this step refer to, or may beperformed by, a region-specific layer component as described withreference to FIG. 3 .

At operation 610, the system generates a region layer based on the firstregion mask and the second region mask and the reference region. In somecases, the operations of this step refer to, or may be performed by, aregion-specific layer component as described with reference to FIG. 3 .

At operation 615, the system generates a composite image by combiningthe first image and the region layer. In some cases, the operations ofthis step refer to, or may be performed by, an image compositioncomponent as described with reference to FIG. 1 .

FIG. 7 shows an example of a region layer diagram according to aspectsof the present disclosure. The example shown includes original image700, hard mask 705, soft mask 710, brushed hard mask 715, brushed softmask 720, and region mask 725. Region mask 725 is an example of, orincludes aspects of, the corresponding element described with referenceto FIG. 9 .

The hard mask 705 and the soft mask 710 may then be manually edited by auser using a mask brush to create a brushed hard mask 715 and a brushedsoft mask 720, respectively. Then, a mask blending process is used tocombine the brushed hard mask 715 and the brushed soft mask 720 into asingle mask. For example, the mask blending process may take a weightedaverage of the two masks. In some cases, user editable parameters maydetermine how the two masks are blended. In some examples, a fade edgeprocess and a shift edge process may be used to blend the brushed hardmask 715 and the brushed soft mask 720. The blending process can be usedto adjust a boundary region (i.e., areas of the resulting mask that arenot binary) in either direction (i.e., to reveal more background or moreforeground). The fade edge process and the shift edge process may alsobe user controllable.

The mask that results from combining and editing the brushed hard mask715 and a brushed soft mask 720 may be referred to as a region mask 725(or a sky layer in the case of sky replacement). The region mask 725also includes a selected region from a reference image (i.e., the imagethat will be masked prior to combining with the original image). In somecases, the reference image may be moved or repositioned (eitherautomatically or dynamically by the user) so that a different portion isvisible through the region mask. In the sky replacement example, thehorizon of the reference image may be positioned to align with thehorizon of the original image as described herein.

According to an embodiment, both global and local edits to the masks cancoexist and be performed in any order without loss of any edit. Localedits (e.g., using the mask brush tool) give fine control to improve theresult and fix defects in the generated masks. Local edits are appliedto copies of the original hard mask 705 and soft mask 710 (e.g., whichmay result in brushed hard mask 715 and brushed soft mask 720). In someexamples, two painted masks are blended together (i.e., according to afade edge setting) to form a combined region mask. Then, a shift edge isapplied to the combined mask. Fade edge and shift edge settings can beadjusted without losing the brush edits (e.g., resulting in region mask725).

Defringing (Lighting) Layer

FIG. 8 shows an example of a lighting layer flowchart according toaspects of the present disclosure. In some examples, these operationsare performed by a system including a processor executing a set of codesto control functional elements of an apparatus. Additionally oralternatively, certain processes are performed using special-purposehardware. Generally, these operations are performed according to themethods and processes described in accordance with aspects of thepresent disclosure. In some cases, the operations described herein arecomposed of various substeps, or are performed in conjunction with otheroperations.

At operation 800, the system identifies a region mask and a lightingmask based on a first image. In some cases, the operations of this steprefer to, or may be performed by, a mask generation network as describedwith reference to FIG. 3 .

At operation 805, the system generates a region layer based on theregion mask and a second image. In some cases, the operations of thisstep refer to, or may be performed by, a region-specific layer componentas described with reference to FIG. 3 .

At operation 810, the system generates a lighting layer based on thelighting mask and the second image. In some cases, the operations ofthis step refer to, or may be performed by, a region-specific layercomponent as described with reference to FIG. 3 .

At operation 815, the system generates a composite image based on thefirst image, the region layer, and the lighting layer. In some cases,the operations of this step refer to, or may be performed by, a layercomposition component as described with reference to FIG. 3 .

FIG. 9 shows an example of a defringing layer diagram according toaspects of the present disclosure. In one embodiment, defringing layer900 includes grayscale version 905 and region mask 910. Region mask 910is an example of, or includes aspects of, the corresponding elementdescribed with reference to FIG. 7 .

FIG. 9 shows an example of a defringing layer diagram according toaspects of the present disclosure. According to certain embodiments, aML algorithm may create a region mask 910 (which represents the extentof lighting from the background region). In some cases, the boundarybetween non-lighting area and defringing region in the defringing layermask corresponds roughly to the boundary between foreground andbackground, but the transition is even more blurry or extended than inthe soft mask.

The region mask 910 may be combined with a grayscale version 905 of thepositioned reference image to generate a defringing layer 900. In somecases, the grayscale version 905 may be arranged between a layerincluding the original image and the replacement region layer (e.g.,mask that results from combining and editing the hard mask and the softmask, or a sky layer in the case of sky replacement) in an image editingapplication.

A machine learning model may also create a lighting layer mask (whichrepresents the extent of lighting from the background region). In somecases, the boundary between a non-lighting area and lighting region inthe lighting layer mask corresponds roughly to the boundary betweenforeground and background, but the transition is even more blurry orextended than in the soft mask.

A lighting mask may be combined with a grayscale version of thepositioned reference image to generate a lighting layer. In someexamples the lighting layer may be arranged between a layer includingthe original image and the replacement region layer (e.g., the mask thatresults from combining and editing the hard mask and the soft mask, or asky layer in the case of sky replacement) in an image editingapplication.

A portion of the sky visible based on the relative position of thereference image and the original image may be used to generatebackground property data (e.g., sky property data). This data may alsodepend on user edits (e.g., edits to the tone, saturation, brightness,or color composition of the sky region of the reference image).Foreground property may also be determined based on the hard or softmask and the original image. The foreground property data and thebackground property data may be used to generate a harmonization layer.The harmonization layer may adjust the color of the foreground so thatit looks more natural with the new background (i.e., so that a landscapewill look more natural with a different sky).

In some examples, a harmonization layer may be arranged between thelighting layer and a layer including the original image in an imageediting application. In other examples, the harmonization layer may bearranged between the replacement region layer and the layer containingthe original image. In some cases, a scaled down version of thereference image and the original image (or the corresponding regions ormasks) may be used when determining the harmonization layer to improvecomputational efficiency.

FIG. 10 shows an example of a process for generating a composite imageaccording to aspects of the present disclosure. In some examples, theseoperations are performed by a system including a processor executing aset of codes to control functional elements of an apparatus.Additionally or alternatively, certain processes are performed usingspecial-purpose hardware. Generally, these operations are performedaccording to the methods and processes described in accordance withaspects of the present disclosure. In some cases, the operationsdescribed herein are composed of various substeps, or are performed inconjunction with other operations.

A method for image editing is described. Embodiments of the method areconfigured to generate a plurality of region masks for a first imageusing a mask generation network, where the region masks correspond to asame semantically related region of the first image. Embodiments of themethod are further configured to generate a defringing layer bycombining a first region mask of the plurality of region masks withgrayscale version of a second image, generate a region-specific layer bycombining a second region mask of the plurality of region masks with thesecond image, and generate a composite image by combining the firstimage, the defringing layer, and the region-specific layer.

At operation 1000, the system generates a set of region masks for afirst image using a mask generation network, where the region maskscorrespond to a same semantically related region of the first image. Insome cases, the operations of this step refer to, or may be performedby, a mask generation network as described with reference to FIG. 3 .

At operation 1005, the system generates a defringing layer by combininga first region mask of the set of region masks with grayscale version ofa second image. In some cases, the operations of this step refer to, ormay be performed by, a defringing component as described with referenceto FIG. 3 .

At operation 1010, the system generates a region-specific layer bycombining a second region mask of the set of region masks with thesecond image. In some cases, the operations of this step refer to, ormay be performed by, a region-specific layer component as described withreference to FIG. 3 .

At operation 1015, the system generates a composite image by combiningthe first image, the defringing layer, and the region-specific layer. Insome cases, the operations of this step refer to, or may be performedby, a layer composition component as described with reference to FIG. 3.

An apparatus for image editing is also described. The apparatus includesa processor, memory in electronic communication with the processor, andinstructions stored in the memory. The instructions are operable tocause the processor to generate a plurality of region masks for a firstimage using a mask generation network, wherein the region maskscorrespond to a same semantically related region of the first image,generate a defringing layer by combining a first region mask of theplurality of region masks with grayscale version of a second image,generate a region-specific layer by combining a second region mask ofthe plurality of region masks with the second image, and generate acomposite image by combining the first image, the defringing layer, andthe region-specific layer.

A non-transitory computer readable medium storing code for image editingis also described. In some examples, the code comprises instructionsexecutable by a processor to: generate a plurality of region masks for afirst image using a mask generation network, wherein the region maskscorrespond to a same semantically related region of the first image,generate a defringing layer by combining a first region mask of theplurality of region masks with grayscale version of a second image,generate a region-specific layer by combining a second region mask ofthe plurality of region masks with the second image, and generate acomposite image by combining the first image, the defringing layer, andthe region-specific layer.

A system for image editing is also described. Embodiments of the systemare configured for generating a plurality of region masks for a firstimage using a mask generation network, wherein the region maskscorrespond to a same semantically related region of the first image,generating a defringing layer by combining a first region mask of theplurality of region masks with grayscale version of a second image,generating a region-specific layer by combining a second region mask ofthe plurality of region masks with the second image, and generating acomposite image by combining the first image, the defringing layer, andthe region-specific layer.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include combining the secondregion mask with a third region mask of the plurality of region masks tocreate a combined region mask, wherein the region-specific layer isgenerated using the combined region mask.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include generating a colorharmonization layer based on at least one of the plurality of regionmasks, wherein the composite image includes the color harmonizationlayer. In some examples, the first region mask has a more gradual maskboundary than the second region mask. In some examples, the defringinglayer is located between the first image and the region-specific layer.Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include adjusting a positionof the second image relative to the first image. Some examples furtherinclude automatically regenerating the composite image based on theadjusted position.

Some examples of the method, apparatus, non-transitory computer readablemedium, and system described above further include editing the compositeimage using an image editing application. Some examples of the method,apparatus, non-transitory computer readable medium, and system describedabove further include selecting the second image from among a pluralityof candidate images for replacing the semantically related region of thefirst image. In some examples, the semantically related region of thefirst image comprises a first sky region, and the composite imagecomprises a second sky region from the second image.

Color Harmonization Layer

FIG. 11 shows an example of a process for generating a composite imageaccording to aspects of the present disclosure. In some examples, theseoperations are performed by a system including a processor executing aset of codes to control functional elements of an apparatus.Additionally or alternatively, certain processes are performed usingspecial-purpose hardware. Generally, these operations are performedaccording to the methods and processes described in accordance withaspects of the present disclosure. In some cases, the operationsdescribed herein are composed of various substeps, or are performed inconjunction with other operations.

At operation 1100, the system generates a foreground region mask for afirst image using a mask generation network. In some cases, theoperations of this step refer to, or may be performed by, a maskgeneration network as described with reference to FIG. 3 .

At operation 1105, the system computes foreground property data based onthe foreground region mask and the first image. In some cases, theoperations of this step refer to, or may be performed by, an imageproperty component as described with reference to FIG. 3 .

At operation 1110, the system computes background property data based ona second image. In some cases, the operations of this step refer to, ormay be performed by, an image property component as described withreference to FIG. 3 .

At operation 1115, the system generates a color harmonization layerbased on the foreground property data and the background property data.In some cases, the operations of this step refer to, or may be performedby, a color harmonization component as described with reference to FIG.3 .

At operation 1120, the system generates a composite image based on thefirst image, the second image, and the color harmonization layer. Insome cases, the operations of this step refer to, or may be performedby, an image editing application as described with reference to FIG. 3 .

FIG. 12 shows an example of a process for color harmonization accordingto aspects of the present disclosure. The example shown includesbackground property data 1200, adjusted background property data 1205,clipped background property mask 1210, masked background property 1215,masked foreground property 1220, foreground harmonization color transfer1225, and foreground color harmonization curves 1230.

According to a sky replacement example, the portion of the sky visiblebased on the relative position of the reference image and the originalimage may be used to generate background property data 1200. Thebackground property may be adjusted to produce adjusted backgroundproperty data 1205. In an example scenario, an adjustment to theposition of the background may produce adjusted background property data1205.

A clip mask may be applied to the background property data 1200 toproduce a clipped background property mask 1210. The adjusted backgroundproperty data 1205 and the clipped background property mask 1210 may beused to create a masked background property 1215.

The background property data 1200 may depend on user edits (e.g., editsto the tone, saturation, brightness, or color composition of the skyregion of the reference image). Masked foreground property 1220 may alsobe determined based on the hard or soft mask and the original image. Themasked foreground property 1220 and the masked background property 1215may be used to generate a harmonization layer comprising foregroundharmonization color transfer 1225 and foreground color harmonizationcurves 1230. The harmonization layer may adjust the color of theforeground so that it looks more natural with the new background (i.e.,so that a landscape will look more natural with a different sky).

In some examples, the harmonization layer may be arranged between thedefringing layer and a layer including the original image in an imageediting application. In other examples the harmonization layer may bearranged between the replacement region layer and the layer containingthe original image. In some cases, a scaled down version of thereference image and the original image (or the corresponding regions ormasks) may be used when determining the harmonization layer to improvecomputational efficiency.

Some embodiments of the present disclosure provide real-timeharmonization based on the visible sky region. When the sky is moved,the change is detected, and the foreground harmonization is appliedaccordingly to show natural composition on canvas.

Preset Loading

Loading high resolution preset images (e.g., images used for skyreplacement) can cause bottlenecks for loading and saving. For example,loading and saving 25 presets can take 10-15 seconds. Furthermore,memory usage can be high when presets are loaded. Accordingly,embodiments of the present disclosure include systems and techniques forpreset loading.

In one example, sky replacement presets are color images of highresolutions (e.g., 6000×4500). As discussed above, this can posechallenges for loading and saving performance, and memory usageefficiency. Traditional preset representation techniques were designedfor patterns, gradients, and styles where the representation data forthe preset types are smaller sizes (e.g., 946×946 for patterns). Thus,FIGS. 13-16 describe efficient techniques for loading preset images(e.g., sky presets).

The representation of a preset comprises multiple files. For example,the metadata and thumbnails for all of the presets can be stored in asingle presets info file, while full resolution image data for thepresets can represented in separate JPEG or PNG files. At runtime, thesingle presets info file including metadata and thumbnails is readinitially, and preset thumbnails are shown for preview. When a preset isselected, full resolution image data of the preset is loaded (i.e., lazyloading), and when another preset is selected, the full resolution imagedata of the previous preset is released from memory. When a preset isdeleted or created, the corresponding JPEG or PNG file is deleted orcreated, and the single presets info file (i.e., the file including themetadata and thumbnails) is updated.

FIG. 13 shows an example of a process for image editing according toaspects of the present disclosure. In some examples, these operationsare performed by a system including a processor executing a set of codesto control functional elements of an apparatus. Additionally oralternatively, certain processes are performed using special-purposehardware. Generally, these operations are performed according to themethods and processes described in accordance with aspects of thepresent disclosure. In some cases, the operations described herein arecomposed of various substeps, or are performed in conjunction with otheroperations.

At operation 1300, the system receives original image data. In somecases, the operations of this step refer to, or may be performed by, apreset component as described with reference to FIGS. 3 and 4 .

At operation 1305, the system retrieves preset information for set ofpreset images, where the preset information for each of the presetimages includes low-resolution image data and region locationinformation. In some cases, the operations of this step refer to, or maybe performed by, a preset component as described with reference to FIGS.3 and 4 .

At operation 1310, the system generates at least one low-resolutionpreview based on the original image data and the preset information. Insome cases, the operations of this step refer to, or may be performedby, a thumbnail component as described with reference to FIG. 4 .

At operation 1315, the system selects a preset image from among the setof preset images based on the at least one low-resolution preview image.In some cases, the operations of this step refer to, or may be performedby, a preview component as described with reference to FIG. 4 .

At operation 1320, the system loads high-resolution image data for theselected preset image. In some cases, the operations of this step referto, or may be performed by, a color conversion component as describedwith reference to FIG. 4 .

FIG. 14 shows an example of a process for image editing according toaspects of the present disclosure. In some examples, these operationsare performed by a system including a processor executing a set of codesto control functional elements of an apparatus. Additionally oralternatively, certain processes are performed using special-purposehardware. Generally, these operations are performed according to themethods and processes described in accordance with aspects of thepresent disclosure. In some cases, the operations described herein arecomposed of various substeps, or are performed in conjunction with otheroperations.

At operation 1400, the system performs a segmentation information oneach of a set of preset images to produce region location information.In some cases, the operations of this step refer to, or may be performedby, a segmentation network as described with reference to FIG. 4 .

At operation 1405, the system generates low-resolution image data foreach of the preset images. In some cases, the operations of this steprefer to, or may be performed by, a thumbnail component as describedwith reference to FIG. 4 .

At operation 1410, the system stores preset information including theregion location information and the low-resolution image data for eachof the set of preset images in a same presets information file. In somecases, the operations of this step refer to, or may be performed by, adatabase as described with reference to FIG. 1 .

At operation 1415, the system stores high-resolution image data for eachof the preset images in separate image files. In some cases, theoperations of this step refer to, or may be performed by, a database asdescribed with reference to FIG. 1 .

FIG. 15 shows an example of a preset image diagram according to aspectsof the present disclosure. The example shown includes source image 1500,color converted image 1535, mask 1510, preset region 1515, thumbnail1520, presets info file 1525, compressed image 1530, and image files1535.

The process of creating presets begins with the generation of a newunique ID saved as a preset ID in a presets info file. In some examples,the unique ID may include or be based on a universally unique ID (UUID).The source image 1500 is processed for color conversion using an RGBcolor space as the default target color profile. The metadata of thecolor converted image is saved in the presets info file 1525.

Additionally, or alternatively, the color converted image is segmentedto create a mask. A region is detected in the mask using an algorithm(for example, an image region with 50-percent bound detection). Thedetected sky region represented by a bounding box in a source image 1500and a thumbnail image created from a color converted image are saved inthe presets info file 1525.

The full resolution data of the color profile converted image iscompressed and saved into an image file 1535, such as a JPEG or PNGfile. In some examples, the source image 1500 is copied into an imagefile 1535 in the presets folder if the format is JPEG or PNG and colorprofile is the same as the default target color profile. The saved imagefile 1535 is named using the preset ID with a suffix (i.e., .jpg or.png). Other image file 1535 are full resolution image files ofpresently used presets.

In some examples, the system receives a subsequent user selection. Forexample, the user may select a different image preset from a userinterface. Then, the system releases the full resolution data of thepreviously selected source image 1500, loads the full resolution data ofthe selected source file as a new replacement image, converts the newreplacement image color profile to the target color profile, detects asky region from the color converted data, and then calculates skyreplacement for a preview using the color converted data of the newreplacement image and its detected sky region.

Thus, embodiments of the present disclosure pre-compute the imageregions and associate the region information with the preset parameters,which may be loaded with other preset info when an image replacementinterface is opened. For the user's new custom preset image, the imageregion may be computed when the new custom preset image is initiallyimported, and then the region information may be associated with the newpreset. Thus, less computation is performed when the preset is used inthe future.

In some cases when loading the presets, only the thumbnail data of thepresets are loaded for preview. The full resolution data of a preset maynot be loaded into the memory until it is selected by the user for imagereplacement. When a new preset image is selected for image replacement,the full resolution data of the previously selected preset image will bereleased from the memory. This preset loading approach providesefficiency of memory usage independent of the number of presets beingpreviewed.

FIG. 16 shows an example of a preset image representation structureaccording to aspects of the present disclosure. The example shownincludes presets info file 1600, preset node 1605, and presetidentification 1610.

The basic structure of a preset image representation comprises presetsinfo file 1600 and one or more full resolution image files stored in adefault location. The presets info file 1600 contains preset node 1605followed by hierarchy info for the presets and groups. The preset node1605 is an instance of a preset containing a preset version ID, presetidentification 1610, metadata, sky region info, and preset thumbnaildata. The preset identification 1610, created as a unique ID, is uniqueand may be used to name the full resolution image file (JPEG or PNG) ofthe associated unique preset. One or more preset node 1605 share apreset identification 1610 and the associated full resolution imagefile. The metadata includes preset source type (for example, default orcustom), image mode, image format, thumbnail image resolution, originalimage resolution, preset name, and color profile. For example, the skyregion info is represented by a bounding box, pre-computed using the skyreplacement segmentation pipeline when the preset is created.

In one example, the preset image representation structure applies topreset images used for preloading information that includes a detectedsky region of an image and loading the full resolution version of theimage based on the user selection.

When a user selects a new preset image, the sky regions in the presetimage may be detected to find the coordinates for placing the new skyinto the composition. In practice, the sky segmentation for a presetimage could take significant computations. Therefore achieving thereal-time sky replacement preview can be challenging. Furthermore, theuser may preview the preset images before selecting. However, since thepreset sky images typically have high resolutions, loading a largenumber of preset images can be slow and can use a significant amount ofmemory. Additionally or alternatively, saving preset image changes canalso be slow if presets are represented in a single file.

To address these issues, the structure of preset image representationmay be configured to contain a single presets info file 1600 and fullresolution image files. The presets info file 1600 contains a list ofpreset nodes and a hierarchy of information of the preset nodes andgroups. Each preset node in the list contains information such as aversion ID, a preset ID, metadata (such as image mode, format, size,color profile, etc.), pre-computed sky region information (e.g., abounding box of the sky region), and thumbnail data.

The full resolution image files contain the compressed image data of thepreset images specified in the presets info file 1600. The base name ofeach image file is the associated preset ID of the preset specified inthe presets info file 1600, and the suffix of each image file is theassociated image format (e.g., .jpg or .png) of the preset specified inthe presets info file 1600.

Preset images may be loaded by reading the presets info file 1600 andrendering the preset thumbnails to preview from the preset node 1605.Preset identification 1610 is then selected. For example, a presetthumbnail may be selected by the user (e.g., via the UI), and then thesystem automatically identifies which preset node is associated with theselected thumbnail.

The full resolution image file name is determined using the preset IDand image format of a currently selected preset image. The fullresolution image file is read, and the full resolution image data isconverted to the target color profile. The preset image data isestablished for replacement with the pre-computed sky region info. Skyreplacement is then calculated. The full resolution image data of thecurrent preset may be released from the memory when a different selectis selected.

The source image file is read and the source image in converted to thedefault preset color profile to create a new custom preset. The skyregion is detected in the source image, and a new unique preset ID isgenerated. The source image is then converted to the default presetimage format (e.g., JPEG or PNG). The converted image file is saved tothe presets folder, and is renamed with the new preset ID. A new presetnode is created, and is added to the current preset node list.

Presets may then be saved by saving the updated preset nodes andhierarchy info to the presets info file 1600. Full resolution images maynot be saved at this time since the full resolution images have alreadybeen temporarily saved when creating new presets. Thus, the savingprocess becomes significantly fast by only updating the presets infofile 1600.

The preset image representation provides increased sky replacement speedby using precomputed sky region info from the selected preset andincreased presets loading speed for previewing. Additionally oralternatively, the preset image representation provides increasedpresets saving speed when there is any update of the presets to persist.Memory is used much more efficiently by loading the full resolutionimage data of the preset upon selection and releasing the fullresolution image data upon deselection.

The workflow of loading presets comprises loading thumbnails andmetadata of presets from the presets info file 1600. For example,loading presets comprises loading thumbnails, region info, and metadatafrom the presets info file 1600. The thumbnails are rendered in a presetview window. The hierarchy of presets and groups is read from thepresets info file and shown in the preset view window. For example, whena preset is selected, the full resolution image of the selected presetis loaded from an associated file (i.e., JPEG/PNG), and the image of thelast selected preset is released. The full resolution image data of thelast selected preset (if any) is released from memory when a preset isselected for image replacement. The image file name is determined usingthe preset identification 1610 and image format of the presentlyselected preset in the preset node 1605. Image data of the selectedpreset is loaded, and the image is converted to a target color profiledetermined by the color profile (e.g., of the original image thatcontains the old sky). In some examples, the preset data is set forimage replacement with pre-computed region info and other metadata andthe image replacement is calculated for preview.

The description and drawings described herein represent exampleconfigurations and do not represent all the implementations within thescope of the claims. For example, the operations and steps may berearranged, combined or otherwise modified. Also, structures and devicesmay be represented in the form of block diagrams to represent therelationship between components and avoid obscuring the describedconcepts. Similar components or features may have the same name but mayhave different reference numbers corresponding to different figures.

Some modifications to the disclosure may be readily apparent to thoseskilled in the art, and the principles defined herein may be applied toother variations without departing from the scope of the disclosure.Thus, the disclosure is not limited to the examples and designsdescribed herein, but is to be accorded the broadest scope consistentwith the principles and novel features disclosed herein.

In this disclosure and the following claims, the word “or” indicates aninclusive list such that, for example, the list of X, Y, or Z means X orY or Z or XY or XZ or YZ or XYZ. Also the phrase “based on” is not usedto represent a closed set of conditions. For example, a step that isdescribed as “based on condition A” may be based on both condition A andcondition B. In other words, the phrase “based on” shall be construed tomean “based at least in part on.” Also, the words “a” or “an” indicate“at least one.”

What is claimed is:
 1. A method comprising: generating a plurality ofregion masks for a first image using a mask generation network, whereinthe region masks correspond to a same semantically related region of thefirst image; generating a defringing layer by combining a first regionmask of the plurality of region masks with grayscale version of a secondimage; generating a region-specific layer by combining a second regionmask of the plurality of region masks with the second image; andgenerating a composite image by combining the first image, thedefringing layer, and the region-specific layer.
 2. The method of claim1, further comprising: combining the second region mask with a thirdregion mask of the plurality of region masks to create a combined regionmask, wherein the region-specific layer is generated using the combinedregion mask.
 3. The method of claim 1, further comprising: generating acolor harmonization layer based on at least one of the plurality ofregion masks, wherein the composite image includes the colorharmonization layer.
 4. The method of claim 1, wherein: the first regionmask has a more gradual mask boundary than the second region mask. 5.The method of claim 1, wherein: the defringing layer is located betweenthe first image and the region-specific layer.
 6. The method of claim 1,further comprising: adjusting a position of the second image relative tothe first image; and automatically regenerating the composite imagebased on the adjusted position.
 7. The method of claim 1, furthercomprising: editing the composite image using an image editingapplication.
 8. The method of claim 1, further comprising: selecting thesecond image from among a plurality of candidate images for replacingthe semantically related region of the first image.
 9. The method ofclaim 1, wherein: the semantically related region of the first imagecomprises a first sky region, and the composite image comprises a secondsky region from the second image.
 10. A method comprising: generating afirst region mask, a second region mask, and a third region maskcorresponding to a same semantically related region of a first image;generating a defringing layer by combining the first region mask withgrayscale version of a second image; generating a region-specific layerby combining the second region mask and the third region mask to producea combined region mask, and combining the combined region mask with thesecond image; and generating a composite image by combining the firstimage, the defringing layer, and the region-specific layer.
 11. Themethod of claim 10, further comprising: applying a mask brush to adjustthe combined region mask; and applying a fade edge adjustment, a shiftedge adjustment, or both the fade edge adjustment and the shift edgeadjustment after applying the mask brush.
 12. The method of claim 11,further comprising: applying the mask brush to further adjust thecombined region mask; and automatically reapplying the fade edgeadjustment, the shift edge adjustment, or both the fade edge adjustmentand the shift edge adjustment after reapplying the mask brush.
 13. Themethod of claim 10, further comprising: generating a color harmonizationlayer based on the second region mask, wherein the composite imageincludes the color harmonization layer.
 14. The method of claim 10,further comprising: adjusting a position of the second image relative tothe first image; automatically regenerating the defringing layer and theregion-specific layer based on the position; and automaticallyregenerating the composite image based on the regenerated defringinglayer and the regenerated region-specific layer.
 15. The method of claim10, wherein: the first region mask has a more gradual mask boundary thanthe second region mask, and the second region mask has a more gradualboundary than the third mask.
 16. The method of claim 10, wherein: thesemantically related region of the first image comprises a first skyregion, and the composite image comprises a second sky region from thesecond image.
 17. An apparatus comprising: a mask generation networkconfigured to generate a plurality of region masks for a first imageusing a mask generation network, wherein the region masks correspond toa same semantically related region of the first image; a defringingcomponent configured to generate a defringing layer by combining a firstregion mask of the plurality of region masks with grayscale version of asecond image; a region-specific layer component configured to generate aregion-specific layer by combining a second region mask of the pluralityof region masks with the second image; and a layer composition componentconfigured to generate a composite image by combining the first image,the defringing layer, and the region-specific layer.
 18. The apparatusof claim 17, wherein: the mask generation network comprises aconvolutional neural network (CNN).
 19. The apparatus of claim 17,wherein: the region-specific layer component comprises a brush tool, afade edge slider, and a shift edge slider.
 20. The apparatus of claim17, further comprising: a color harmonization component configured togenerate a color harmonization layer based on at least one of theplurality of region masks.